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Introduction 


Throughout the literature authors have consistently discussed the 
suspicion that regression results were less than satisfactory when 
the independent variables were correlated. Camm, Gulledge, and 
Womer [1], and Womer and Marcotte [2] provide excellent applied 
examples of these concerns. Many authors have obtained partial 
solutions for this problem as discussed by Womer and Marcotte [2] 
and Wonnacott and Wonnacott [3], which result in generalized least 
squares algorithms to solve restrictive cases. 

This paper presents a simple but relatively general multivariate 
method for obtaining linear least squares coefficients which are 
free of the statistical distortion created by correlated independent 
variables. 

The Method 


The multivariate linear least squares problem is stated as 



2 


minimize ^ 8 k 2 

k 

with respect to a and e k 

subject to e k = y k -x~ k T a for k = 1 ... n, 

where y k is the kth measurement of the dependent variable, a is the 

vector of m desired parameters a,, x k is the kth measurement of the 
vector 


x = 




V X m J 


of the independent variables x, with x ok = 1 , 


x k T =(x k o-x k m ) 


is the transpose of x k , and e k is the kth value of the error 8. 

The problem is called linear since the model 

8 = y - x~ T a 

is linear in the parameter vector a. Setting x ok = 1 provides a 
constant term a 0 in the model. 

Note that the linear least squares problem exists as an optimization 
problem independent of statistics. It also has a statistical 
interpretation when either the independent variables x k or the error 
8 k are considered as random variables. In this context the error is 
often called the residual. 

Traditional econometric methods apply statistics to the analysis of 
the residuals. The foundations for this paper are based on a second 
approach found in Fukunaga [4], He assumes that the data 
observations themselves are random variables. The independent 
variables in aerospace parametrics are typically observations of 
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possible projects as measured by system requirement, system 
performance, system design, or engineering process metrics. Since 
the finished project is only one of the many possible projects which 
could have been completed [5,6], the data observations are 
themselves drawn from the distributions of the measures of the 
possible projects. Thus Fukunaga's paradigm applies. 

In classical linear least squares as described by Draper and Smith 
[7] we let y be the vector of the n dependent variable observations 
y k , 8 be the vector of the n errors £ k , and 


/A t A 


X = 


x 




be the vector of n independent variable observation transpose 
vectors x~ k T , the problem may be restated in vector form as 


minimize e T e = (y - X a ) T (y - X a ). 


Setting the partial derivatives with respect to a to zero we have the 
normal equations 


X T X a = X T y 

where a~ is the estimate of a 

Noting that X T X is a m by m matrix, which with at least m distinct 
data points should have full rank, we assume that X T X may be 
inverted to obtain 

a = (X T X)- 1 X T y . 

It is important to note that, when the x k are not considered random 
variables, the components a j are linear functions of the random 
dependent observations y k . Under this interpretation, a provides a 
statistically unbiased estimate of a which has the minimum 
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variance of all linear unbiased estimators of a, irrespective of 
distribution properties of the errors. 

However, when the x are considered random variables, the 
components aj are no longer linear functions of random variables. 
Following Papoulis [8] we use the property of conditional probability 
distributions that for independent random variables Zj 

E{g(z) | z } = g(z-) 
to obtain 

E{a | x , 8} = E{(X t X)- 1 X T y | x , 8} = (X T X)" 1 X T y = a 

since y is a function of x and 8 Thus if the random variables x and 
the errors 8 are statistically independent a is still an unbiased 
estimate of a - . 

Unfortunately, statistical independence is a strong condition in 
practice since many of the metrics used in cost analysis are highly 
correlated. Thus it is desired to obtain coefficients aj which are 
free of the statistical distortion caused by performing a linear least 
squares fit when there is correlation between the x,. 

A general method follows through which the coefficients a, may be 
found by first transforming the x, into a new set of random variables 
Zj, performing the least squares fit on the Zj, and then transforming 
the coefficients bj found by the least squares process to obtain the 
desired coefficients aj. 

Following Fukunaga [4] an uncorrelated and if normally distributed a 
statistically independent set of random variables may be obtained 
using the eigenvalues and eigenvectors of the covariance matrix 

H x = E{(x-p x )(x~-p x ) T } 


= E{(Xj-p x .)(x r [i x .)} 
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( ( x rM'x 1 )( x rM'Xi) 


( X 1 ^x-|)( x m HxJ 


V 


( x m ^x m )( x 1 tV,) ■■■ ( x m ^x m )( x m fT< m ) 


A 

) 


The eigenvectors of the covariance matrix H x are directions of 
statistically independent random variables and the eigenvalues are 

the associated variances. The eigenvectors to, are called principal 
components in statistical jargon. Figure 6.1 illustrates the 
principal component axes formed in two dimensions by correlated 

variables x-, and x 2 . The vector co-, indicates the direction of 

maximum variance a-, 2 . The vector co 2 indicates the direction of 
minimum variance o 2 2 . Since the principal component vectors have 
unit magnitude by definition, the standard deviations may be 
represented as standard deviation vectors with origin translated to 
sample mean. Thus 


o-| = o-| co i and 
o 2 = o 2 co 2 . 



Figure 1 

Principal Component Axes and Standard Deviation Vectors 

When the distributions of the x, are normal, the standard deviation 
vectors multiplied by the same constant 6 form the axes of an 
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ellipse representing a 3 standard deviation equal probability 
contour. 

Let A,jj = Gj 2 be the ith eigenvalue of H x and co, be the eigenvector of H x 
associated with X, jj. Then there exists a diagonal matrix A with the 

Xjj as diagonal components, a matrix Q with columns coj, and the 
transpose of Q such that 

H x Q = Q A and Q Q = I . 

Thus H x may be expressed as 

H x = Q AQ t . 

Form new random variables z, by the rule 
z = A x . 

The covariance matrix H z of the Zj has the form 
H z = A H x A t . 

The square root matrix M 1/2 is defined by the property 
M 1/2 M 1/2 = M 
with inverse M _1/2 . 

Letting 


A = A - 1/2 C2 t 


then 

H z = A“ 1/2 H x Q AT 112 = A~ m a A C2 T Q. A~ m = I. 

The transform rotates the axes to coincide with the direction of 
the principal components. The transform A _1/2 divides the new basis 
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vector magnitudes a\ by a\ to provide unit basis vector magnitudes 
coinciding exactly with the principal components. 

Note that the standard deviation vectors of Figure 1 have equal 
magnitude when the variances are equal. Thus for a normal joint 
uncorrelated distribution the equal probability contours form a 
circle. This transformation is called a whitening transform since 
the transformed variables are representative of white (totally 
uncorrelated normal) noise. 

The random variables z, resulting from the transform are not only 
uncorrelated since the covariance matrix is diagonal, but are also 
statistically independent if the x, are normally distributed. 

The problem to be solved now is 

minimize ^ 8 k 2 

k 

with respect to b and the e k 

subject to e k = y k -z~ k T b for k = 1 ... n. 

The composite transformation provides random variables 

z = A~ m x 
and restates the problem as 
minimize ^ 8 k 2 

k 

with respect to b and the £ k 

subject to: yk - (A“ 1/2 x k ) T b = £ k for k = 1 ... n, 

By setting 


a = Q A 1/2 b 
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we have a solution in terms of the untransformed data. 

Noting that 

z = x a a - 1/2 

we have 

a = Q. A ~ 112 B 
= a A~ m (Z T Z )" 1 Z T y 

= C2 A _1/2 ((A “ 1/2 qJ X t ) (X a A - 172 ))- 1 (A - 1/2 C2 t X t ) y 
a = a A - 1/2 (A 1/2 a T (X T X)- 1 C2 A 1/2 ) A - 1/2 X T y 

= a (A -1/2 A 1/2 ) C2 T (X T X)- 1 a (A 1/2 A _1/2 ) Cl J X t y 
= (^^(XTX)- 1 (C2C2 t )X t y 
= (X T X) _1 X T y . 


Thus the parameter a is a solution to the original problem. 

A simple computer algorithm for implementing this method is as 
follows. 

Input the data vectors x k . Calculate the covariance matrix H x . Find 
the eigenvalues and eigenvectors of H x . Generate the transform 
matrix 


A = A - 1/2 C2 t . 

Transform each data vector x k by A to obtain z k . Apply conventional 

linear least squares to the data (y^, z k ) to obtain the coefficients bj. 
Use 
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a = A t b = Q. A 1/2 B 

to transform the coefficients bj by A T to obtain the desired 
coefficients aj. 

Note that a is an unbiased estimate of a since a is a linear 
transform of b which is an unbiased estimate of b 

Wilkinson [9] demonstrates many methods for finding eigenvalues 
and eigenvectors. An efficient algorithm based on Hildebrand [10] 
coded in Pascal may be found in Flanders [11]. Multivariate least 
squares algorithms in Pascal may be found in Miller [12] which may 
be combined with the eigenvalues/eigenvector procedures to 
generate your own custom software for implementing this technique. 

Conclusion 


Assuming that the independent variables x, are random variables, 
which is representative of most parametric cost applications, the 
linear least squares coefficients a, obtained by the method in this 
paper are free of the statistical distortion from a linear least 
squares fit over the correlated independent variables x,. 

The result is that the analyst can use this technique without concern 
for colinearity or correlation of the independent variables. 

Although, only one transformation has been discussed in this paper, 
there exists a class of transformations which will yield the same 
freedom from statistical colinearity distortion. 

Finally the analyst should also note that they may perform any least 
squares technique they normally use, such as stepwise regression, 

to obtain the b, with the data (y^, z k ). The a, are still found from the 
transformation 

a = Q. A ~ m b. 
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